To test the theoretical expectations discussed in the previous section, we designed an online randomized survey experimentFootnote 11 and administered it in Japan from April 26 to May 2, 2017 on CrowdWorks, a crowdsourcing marketplace in Japan similar to Amazon Mechanical Turk where participating workers complete tasks for monetary compensation.Footnote 12 A total of 3,198 Japanese citizens of voting age (at least 18 years old) completed the survey.Footnote 13

Treatment Variables

After an initial set of questions about survey-takers’ political attitudes,Footnote 14 participants were randomly assigned with equal probability to one of eight treatment conditions, which involved reading a two-sentence policy message. While Bullock (2011) uses substantial content size for his treatment message (newspaper articles with 627–647 words) and demonstrates its significant consequences, our messages are shorter, amounting to roughly 40 words each. An advantage of Bullock’s construction, which contains detailed and unambiguous messages, is that it could administer the policy information treatment to subjects more strongly. Yet, as other scholars (e.g., Boudreau and MacKenzie 2014) note, given that people are more commonly exposed to relatively brief policy information such as through news article excerpts or headlines, presenting a shorter message in our experiment grants a greater degree of external validity. This is particularly relevant in our study because Trump sends messages to global audiences through his tweets, which can be only up to 280 characters.Footnote 15

The message manipulated in our experiment varies by source cue (Trump or an anonymous U.S. Congressman), policy content (cooperative or uncooperative), and issue salience (security, high salience; or educational/cultural exchange programs, low salience):

Cooperative × Exchange Program “[U.S. President Donald Trump / A U.S. Congressman] stated the U.S. should strengthen educational and cultural exchange programs with Japan and applauded past U.S. budget spending to promote mutual understandings with these countries. He also said that such programs should foster trustful relations with foreign countries.”

Cooperative × Security “[U.S. President Donald Trump / A U.S. Congressman] stated that the U.S. should help Japan with paying for its own protection and applauded past U.S. defense spending for the protection of Japan. He also said the U.S. should maintain defense cooperation with Japan.”

Uncooperative × Exchange Program : “[U.S. President Donald Trump / A U.S. Congressman] stated that the U.S. should abolish educational and cultural exchange programs with Japan and denounced past U.S. budget spending to promote mutual understandings. He also said that the country’s budget should rather be used for American people.”

Uncooperative × Security: “[U.S. President Donald Trump / A U.S. Congressman] stated that Japan should pay entirely for its own protection and denounced past U.S. defense spending for the protection of Japan. He also said that the U.S. should not get involved in Japan’s defense policy.”

The statements used in our experiment, particularly those with uncooperative policy content, were slightly altered but still based on actual statements made by Trump during his presidential campaign,Footnote 16 or constructed from other pertinent information such as language used in a budget proposal from his administration.Footnote 17 In other cases, the statements are tied to actual statements but modified; specifically, for changing a statement from an uncooperative policy approach to a cooperative one.

Some further justifications and explanations for our treatments are in order.

Source Cue Treatment Construction

Two factors motivated our selection of an anonymous “U.S. Congressman” as the appropriate comparison case to Trump in designing the source cue treatment. First, we attempted to produce a generic baseline politician that would not prime any preexisting positive considerations (e.g., through the use of former President Barack Obama) or negative ones. This serves to better isolate the effect of Trump. Second, we aimed to minimize connections to Trump (e.g., the Vice President Mike Pence, or any leader of the Republican Party) to reduce the potential for respondents to think of Trump when we do not want them to do so in this source cue control condition. Our comparison choice makes it clear that it is a political figure distinct from the presidency but still from the U.S. Using the U.S. administration, president (without Trump’s name), leader, or other non-presidential politicians still associated with the current presidency could result in survey-takers conflating this figure with Trump and therefore undermine the experimental manipulation.Footnote 18

Tone Versus Content of Policy Messages

We did not further decompose the effect of policy content into the effect of a tone and the effect of specific policy content, because when a policy is cooperative (uncooperative)—namely, when study participants are likely to perceive that a policy will have a positive (negative) impact on their country—it is unlikely that the policy message is delivered with an uncooperative (a cooperative) tone. We could have designed treatments with a more neutral tone with varying direction, but such treatments could have made the content of a policy and its consequences more ambiguous. More importantly, Trump’s policy statements tend not to be neutral. For these reasons, our use of treatments with either cooperative or uncooperative policy content seems justifiable from a perspective of external validity.

Validity in Source Cue × Policy Content Treatments

In our design, we included the Source Cue × Policy Content interaction to examine whether the effect of Trump attribution is conditional on uncooperative policy content. Some might view one of the resulting combinations—Trump attribution and cooperative policy—as a non-credible vignette for survey-takers, which could undermine the experiment’s validity. This represents a potential issue with our experimental design, in which participants were asked to read a short hypothetical policy message but still expected to reflect a plausible real world scenario. However, patterns in Trump’s actual rhetoric should relieve much of this concern.

A few months before fielding our experiment, Japanese Prime Minister Shinzo Abe visited Trump in a meeting that made international news and received much attention in Japan.Footnote 19The New York Times, for example, reported that Trump pledged close security cooperation with Japan, and during this visit, Trump said the following: “We are committed to the security of Japan ... and to further strengthening our very crucial alliance.” This exemplifies cooperative messages from Trump toward Japan having occurred in the real world. Japanese individuals can still overwhelmingly view Trump and his messages with disdain, as public opinion data would indicate (e.g., Wike et al. 2016), but they may also recognize that he can take either hostile or friendly stances toward Japan. Furthermore, this inconsistency in Trump’s policy approach toward Japan fits well with his known history of vacillating on several policy areas,Footnote 20 a habit that the public (in and outside the U.S.) may observe.

A related concern is that uncooperative U.S. policy messages—even from an anonymous source—could evoke images of Trump for Japanese individuals in a way that makes the two treatments, Source Cue and Policy Content, inseparable in the minds of study participants. If this were true, then Japanese citizens should react to an uncooperative message in a similar manner regardless of whether the messenger was explicitly mentioned as Donald Trump or an anonymous congressman. But this is an empirical question, which we examine in the “Results” section.

Explanation of Issue Salience Manipulation

Accounting for issue salience in studies of the relative strength of source cues and policy content is difficult because salience largely cannot be manipulated at the level of individual study participants (but see Mullinix 2016). Instead, we can only manipulate issue areas with varying levels of importance to national political debate as, for example, Arceneaux (2008) does effectively. Along similar lines, as Ciuk and Yost (2016) describe, we can assume that some issues are “hot” on the political agenda and receive more attention than others. On these high salient issues, as compared to lower salient ones, individuals are more often exposed to relevant information and thus retain more considerations about the issue in recent memory when probed for their opinion (Zaller 1992).

In our experimental design, national security represents an issue at the center of public attention in Japan. For example, security policy played a key role in Japan’s recent 2017 election, with Prime Minster Abe making his national security track record central to his campaigns.Footnote 21 While U.S.–Japan exchange programs are not entirely unknown among Japanese citizens, it is highly unlikely that they match national security issues in the space they occupy in the national “political agenda.” Therefore, we argue that our manipulation of issue salience should be effective in terms of general prominence in national attention and discourse.

We also note that our study in Japan coincided with a period of heightened security concerns in the East Asian region, with rhetoric from the U.S. escalating possible conflict.Footnote 22 Not only should the issue of security become further salient for our study’s subjects, but it should also carry a negative connotation in the context of Japanese evaluations of the U.S. Thus, all else being equal, exposure to the U.S. taking a position on security—compared to taking a position on a less salient issue like exchange programs—should engender more negative Japanese views of the U.S. given these recent U.S.-security considerations in memory.

Outcome Variable

The outcome variables are based on responses to questions about perceptions of the U.S. To capture the multidimensional aspects of the U.S. image in the minds of foreign individuals, we asked four questions using five-point, bipolar Likert scales: “Do you have a favorable or unfavorable view of [the United States, United States foreign policy toward Japan, Americans, or Donald Trump]?” The same scale ensures consistency and ease of comparison across all the outcome variables.Footnote 23 This type of survey question about favorability has been regularly asked in one of the most prominent global public opinion surveys, the Pew Global Attitudes Project.Footnote 24 Because we are interested in capturing general attitudes toward the U.S. among Japanese citizens, we use a composite index created from averaging the four questions, which encapsulates the different dimensions of foreign opinion toward the U.S. The pairwise correlation coefficients between each question scale are all positive and range from 0.23 to 0.50. Given that these are all in the same direction and moderately correlated, we consider averaging across the four variables to be a sensible approach.Footnote 25

In additional analysis, we examine these four perception variables independently, present all the results in the Supplementary Materials, and summarize them in “Results” section. We are reluctant to delve further into this inquiry because we do not have good theoretical expectations for the effects on specific attitudes toward the U.S.

Hypotheses

In our statistical analysis, we use the degree of unfavorability as our outcome variable, y, ranging from 1 (most favorable) to 5 (most unfavorable). There are three dichotomous treatment variables: Source Cue takes on a value of 1 for a Trump attribution and 0 for a U.S. Congressman attribution, Policy Content takes on a value of 1 for uncooperative policy content and 0 for cooperative policy content, and Issue Salience takes on a value of 1 for a security issue and 0 for an exchange program issue. Given these variables, we test three preregistered hypotheses using ordinary least squares (OLS) regressions. Below, we introduce two main hypotheses. The third hypothesis and the results of testing it are presented in the Supplementary Materials.

The first hypothesis is based on a simple additive model:

Hypothesis 1

In an additive model, \(y = b_0 + b_1 \cdot \hbox {Source Cue} + b_2 \cdot \hbox {Policy Content} + b_3 \cdot \hbox {Issue Salience} + \epsilon\), the estimated coefficients for the three treatment variables, \(\hat{b_1}\), \(\hat{b_2}\), and \(\hat{b_3}\) are all positive (where, again, higher values on the outcome correspond to more negative views).

The source cue effect (\(b_1\)) is motivated by existing evidence pointing to unfavorable Japanese views toward Trump, and the importance of source cues and associated source credibility level (e.g., Chong and Druckman 2007; Nicholson 2012; Wike et al. 2016). When Japanese study participants have this source cue available, their negative attitudes toward Trump should be more likely to be activated and consequently affect their opinions toward the U.S.

The policy content effect (\(b_2\)) draws from past research showing the potency of threatening messages and information for shaping public opinion (e.g., Klar 2013; Brader 2006; Miller and Krosnick 2004). Thus, those who are exposed to an uncooperative statement (relative to those exposed to a cooperative statement) are more likely to express greater unfavorability toward the U.S.

The issue salience effect (\(b_3\)) is informed by past research showing that issue salience is relevant in the study of foreign public opinion (Goldsmith and Horiuchi 2012). We examine the effect of issue salience itself as an exploratory check, as no study has tested whether it has an independent effect on public opinion. Because the study was undertaken after Trump became the U.S. president and international security issues under his presidency have often had negative connotations, we expected that study participants would tend to build some negative images of the U.S. when they were exposed to information about this contentious topic, regardless of direction of the policy content. Similarly, as discussed in “Treatment and Variables” section, U.S.–security rhetoric around the time of our experiment should lead to more negative Japanese reactions in this domain. Thus, exposure to a U.S. figure taking a position on this fraught issue could possibly move opinion. However, existing research suggests that it is more relevant as a moderator.

We also estimated a two-way interaction model, in which the main variable of our interest—Source Cue—is interacted with the two other treatment variables:

Hypothesis 2

In a two-way interaction model, \(y = c_0 + c_1 \cdot \hbox {Source Cue} + c_2 \cdot \hbox {Policy Content} + c_3 \cdot \hbox {Issue Salience} + c_4 \cdot \hbox {Source Cue} \cdot \hbox {Policy Content} + c_5 \cdot \hbox {Source Cue} \cdot \hbox {Issue Salience} + \epsilon\), the estimated coefficients for the two interaction variables, \(\hat{c_4}\) and \(\hat{c_5}\), are nil, while \(\hat{c_1}\) is positive.

Although some studies suggest interactions of various factors shape American (Ciuk and Yost 2016) and foreign (Goldsmith and Horiuchi 2012) public opinion, in the context of our study, the effect of Trump attribution is expected to be so strong that it does not vary across type of policy content or issue area. This means that the average level of unfavorable attitudes is indistinguishable between Trump\(\times\)Security and Trump\(\times\)Exchange Program conditions. Similarly, the average should not be different between Trump\(\times\)Uncooperative and Trump\(\times\)Cooperative conditions. In other words, the source effect of Trump attribution outweighs all other factors.

Strong past evidence of people predominantly relying on source cues over policy content in opinion formation (as summarized by Bullock 2011) motivates this hypothesis. Moreover, the sharp drop in the percentage of favorable attitudes toward the U.S. president in 2017, as described in “Theoretical Expectations” section, further suggests an especially powerful Trump influence on Japanese opinion. Although more recent work in American politics argues for a stronger effect of policy content, strongly negative images of Trump in the minds of Japanese people (Wike et al. 2016) led us to believe that the source cue and its credibility would be overwhelmingly important for Japanese opinion formation.