How do we quantify polarization?

Lessons from American Presidential Candidates

Alex Zhao

Introduction

In the United States, controversy across the political spectrum often centers on a perceived increase in polarization and a lack of substantive policy analysis by candidates. Interestingly, research suggests that people who voted for Donald Trump in 2016 are unlikely to change their minds regardless of facts and filter all new information through a partisan lens, confirming suspicions of rampant polarization (Aschwanden, 2016). It is thus crucial to have an objective and consistent metric to analyze distinctions between candidates and their policy positions, especially as the 2020 election looms.

In the US, polarization is the separation between the Democratic Party and the Republican Party on policy positions. In order to test levels of polarization in the current elections, we can compare differences between the two parties on specific policy issues. I chose two categories — foreign policy and domestic policy. The foreign/domestic distinction is relatively clear-cut and may reveal insights about the modern party system. We thus end up with two hypotheses:

Hypothesis 1: Democratic candidates and Republican candidates significantly differ in their integrative complexity levels on foreign policy.

Hypothesis 2: Democratic candidates and Republican candidates significantly differ in their integrative complexity levels on domestic policy.

Integrative complexity (IC) is a measurement of information processing and the “elaboration and complexity of any given information or thought” (Thoemmes and Conway, 2007). These thoughts must be captured in a way that can be practically evaluated — for instance, spoken word or written language of any kind. IC can be defined by two primary elements: differentiation and integration. Differentiation, which is a prerequisite to integration, is the perception of multiple ideas, alternatives, or approaches to thinking about an issue. Integration is the perception of relationships between these ideas. Importantly, integrative complexity is not concerned with the content of the speaker’s argument, but its structure; complexity is thus not directly tied to political orientation in any way.

Extensive research has been done on the relationship between complexity and effective leadership, including studies on the Cuban Missile Crisis, surprise attacks, and peace negotiations (Guttieri et al., 1995; Suedfeld and Bluck, 1988; Tibon, 2000); however, research has been lacking regarding the distinctions in IC between American politicians. The purpose of this study is to analyze presidential candidates’ integrative complexity based on a set of interview transcripts. In contrast to previous research, which has focused on prepared speeches and addresses, this study analyzes 77 interviews from presidential candidates for complexity. I test for distinctions between Republican and Democrat presidential candidates and differences in complexity between parties on foreign policy and domestic policy.

Literature Review

As a metric of information processing, integrative complexity (IC) has been applied to a broad range of cases involving political decision-making and leadership (Conway, Suedfeld, & Clements, 2003). Conway, Suedfeld, & Tetlock (2001) found that a drop in IC was associated with an increase in the likelihood of violent conflict, while high IC among political leaders predicted peaceful resolution. Leaders with lower levels of complex thinking were more likely to resort to military action, rather than compromise. Several other studies further suggest that IC levels are extremely relevant for explaining the policy outcomes of leaders (Suedfeld, Corteen, & McCormick, 1986; Tetlock, Peterson, & Berry, 1993). By analyzing transcripts of interactions among key decision-makers during the Cuban Missile Crisis, Guttieri et al. (1995) found that American officials maintained moderate levels of IC; mean IC scores increased when leaders were in private, implying that the intense scrutiny of the public eye suppressed complexity.

As a group, presidential candidates have been thoroughly studied by psychologists and political scientists alike. Thoemmes and Conway (2007) identify both intrinsic and extrinsic reasons for studying presidents: not only are they particularly interesting due to their monumental achievement, they also function as symbols for the pinnacle of political success and thus can also serve as subjects for the study of the party system at large. Learning about the candidates may allow us to extrapolate information about the party’s message, the changing goals of party members, and the structure of the parties themselves.

Division of Democrat and Republican Party members from 1989 to 2011. Source: Andris et al

Beinart (2008) traces the development of party divisions in the context of foreign policy from 1948 onward. In 1947, Senator Vandenberg asserted that “politics end at the water’s edge”; in other words, that the political turmoil and divisiveness of American government should not apply to foreign policy because the parties must represent a united front to other countries. For much of the Cold War, there existed a broad bipartisan consensus on the overall strategy for U.S. foreign policy. However, as time went on, divisions between the two major parties became more pronounced and obvious. From Vietnam to the War on Terror, Democrats and Republicans gradually drifted apart. The War on Terror began as a nearly unanimous effort, with the PATRIOT Act and campaign in Afghanistan supported by all but one Democrat. In contrast, the U.S. invasion of Iraq has become highly divisive issue that characterizes the difference between Democrats and Republicans on foreign policy. Democrats are now framed as “doves” that are generally soft in foreign affairs, while Republicans are the “hawks.” Of course, there are exceptions (the libertarian wing in particular comes to mind). In general, though, Republicans have become more hawkish and likely to support military force; the Pew Research Center (2005) found that the partisan gap on military versus diplomatic methods to keep the peace grew from 16 percent in 1999 to 44 percent in 2004.

Materials and Methods

Previous studies of presidents and politicians have included State of the Union addresses and other prepared speeches (Thoemes and Conway, 2007; Tetlock, 1981). In this paper, we have chosen to analyze interview transcripts for a few reasons. First, prepared speeches may not be crafted by the candidate. Politicians often heavily rely on speechwriters for assistance, thus making IC an analysis of the candidate’s staff rather than the politician herself (Einhorn, 1982). Second, speeches are usually prepared relatively far in advance, while interviews force a greater degree of spontaneous thinking. Thus, interviews may be a better representation of candidates’ complexity than speeches.

I used interviews from Republican and Democratic candidates from the 2016 election cycle that stretch back to July 2015, with the exception of Sarah Palin’s interviews from 2008. Palin’s interviews were included to improve representation of Republican women in the data. The transcripts were stripped of all metadata and interviewer participation, containing only the candidate’s words. Only candidates included in official party debates were studied, since other politicians did not engage in extensive interviewing. Four groups of trained coders graded up to 35 transcripts. Each interview was split up into segments of 150 words each. I categorized each segment as dealing with foreign policy, domestic policy, or neither. Segments that discussed neither foreign nor domestic policy were excluded from my analysis of topic-specific complexity, but included in the evaluation of candidates’ IC overall.

As per convention in qualitative analysis, each group randomly selected five segments from each interview for coding. Individual coders within groups analyzed interviews with no knowledge of other coders’ scores to minimize intercoder influence, with the exception of a few interviews.

Each column represents a different measure of intercoder reliability (i.e. how much we can trust their scores).

Each group calculated its average IC score for each candidate. I then weighted each group’s average based on the ratio of its Krippendorff’s alpha, a measure of reliability, to the average alpha value. The average value of Krippendorff’s alpha was .65. Groups with a value above this average received a higher weight, while groups with a value below .65 received less weight when calculating the final IC scores.

To determine whether the difference between Republican and Democrat candidates on policy positions is statistically significant, I used an unpaired t-test. I treated party affiliation (Democrat, Republican) as the independent variable and IC score as the dependent variable. Group 1 is the Democratic Party and Group 2 the Republican.

Results and Discussion

Weighted averages of each candidate can be found below. Democratic candidates had significantly higher scores than Republican candidates. When comparing aggregate Democratic and Republican scores, Democrats were, on average, 0.55 higher on the scale. Democrats average 2.19 and Republicans 1.64.

Error bars represent standard deviation, n is the number of segments analyzed. For instance, Ben Carson had scored 1.75 on average, with a standard deviation of 0.89 and 50 distinct segments.

Figure 3 displays the mean IC scores, segmented into party affiliation and policy topic (foreign versus domestic policy). Even a cursory glance at the results seems to confirm both hypotheses. Democrats have higher mean scores on both foreign and domestic policy. However, direct comparison of aggregate IC values does not demonstrate a true relationship.

Figure 3: Mean complexity by party and policy issue.

After segmenting for the foreign/domestic policy divide, t-test results (see Figure 4) show a weak correlation in the arena of domestic policy (p = 0.054). Although not statistically significant, this may show that Democrats are only marginally more complex on issues of domestic policy.

On the other hand, in foreign policy, the difference between the two parties is both larger and extremely statistically significant. With a p-value of less than 0.0001, the t-test shows that Democrats are, on average, 1.01 higher on the complexity scale.