Sample

Between May and August 2009, 74 adults were recruited to participate in semi-structured in-depth elicitation interviews that lasted an average of 43 minutes (ranging from 16 to 124 minutes) and included the presentation of a public health framed essay on climate change. The recruitment process was designed to yield completed interviews with a demographically and geographically diverse group of at least 10 people from each of the previously identified "Six Americas" [7]. Four respondents were dropped from this study due to incomplete data, leaving a sample size of 70. Audience segment status (i.e., which one of the "Six Americas" a person belonged) was assessed with a previously developed 15-item screening questionnaire that identifies segment status with 80% accuracy [unpublished data].

To achieve demographic diversity in the sample, we recruited an approximately balanced number of men and women, and an approximately balanced number of younger (18 to 30), middle-aged (31 to 50), and older (51 and older) adults (see Table 1). We did not set recruitment quotas for racial/ethnic groups, but did make an effort to recruit a mix of people from various racial/ethnic backgrounds.

Table 1 Distribution of Respondents by Age, Gender and Segment. Full size table

To achieve geographic diversity, we recruited participants in one of two ways. The majority of participants (n = 56) were recruited - and then interviewed - face-to-face in one of two locations: out-of-town visitors were interviewed at a central location on the National Mall in Washington, DC (a national park situated between the U.S. Capitol, the Smithsonian Museum buildings, and the Lincoln Memorial); and shoppers were interviewed at an "outlet" mall (i.e., discount branded merchandise shopping mall) adjacent to an interstate freeway in Hagerstown, MD. The outlet mall is more than an hour driving distance outside of Washington, DC and attracts shoppers from Maryland, Pennsylvania, and West Virginia, as well as visitors from further away who are driving the interstate freeway. The remaining study participants were recruited via email from among participants to a nationally representative survey that we conducted in Fall 2008 [7]. They were interviewed subsequently by telephone, after being mailed a copy of the test "public health essay" - described below - in a sealed envelope marked "do not open until asked to do so by the interviewer." As an incentive to participate, all respondents were given a $50 gift card upon completion of their interview. George Mason University Human Subjects Review Board provided approval for the study protocol (reference #6161); all potential respondents received written consent information prior to participation.

The 70 study participants resided in 29 states. Using U.S. Census Bureau classifications, 14% (n = 10) were from the Northeast region, 21% (n = 15) were from the Midwest, 40% (n = 28) from the South, and 23% (n = 16) were from the West; state and region were unknown for one participant. In 2006, the geographic distribution of the overall U.S. population was 18%, 22%, 36% and 23% in the Northeast, Midwest, South and West, respectively [23].

Data Collection and Coding

The majority of the interview was devoted to open-ended questions intended to establish the respondent's emotions, attitudes, beliefs, knowledge and behavior relative to global warming's causes and consequences. For example, respective open-ended questions asked alternatively if, how, and for whom global warming was a problem; how global warming is caused; if and how global warming can be stopped or limited; and what, if anything, an individual could do to help limit global warming. Toward the end of the interview, respondents were asked to read "a brief essay about global warming" (see Appendix 1), which was designed to frame climate change as a human health issue. Respondents were also given a green and a pink highlighting pen and asked to "use the green highlighter pen to mark any portions of the essay that you feel are especially clear or helpful, and use the pink highlighter pen to mark any portions of the essay that are particularly confusing or unhelpful."

As shown in Appendix 1, the one page essay was organized into four sections: an opening paragraph that introduced the public health frame (5 total sentences); a paragraph that emphasized how human health will be harmed if action is not taken to stop, limit, and/or protect against global warming (i.e., a description of the threat; 7 sentences); a paragraph that discussed several mitigation-focused policy actions and their human health-related benefits if adopted (4 sentences); and a brief concluding paragraph intended to reinforce the public health frame (2 sentences).

When respondents finished the reading, they were asked to describe in an open-ended format their "general reaction to this essay." (Note: This question was inadvertently not asked of one respondent, therefore the sample size for analysis of this data is 69.) For each portion of the essay they marked in green, they were subsequently asked: "What about each of these sentences was especially clear or helpful to you?" For each portion of the essay they marked in pink, they were also asked: "What about each of these sentences was especially confusing or unhelpful to you?"

To evaluate the respondent's general reactions to the essay we reviewed their individual statements (n = 193), defined as discrete thoughts or concepts. Based on this review, we iteratively developed eight thematic categories that captured the range of statements made by respondents. Table 2 defines and describe these themes.

Table 2 Thematic Categories Used to Code Respondents' General Reactions to the Public Health Essay. Full size table

Two graduate student coders were then trained to code each statement into one of the thematic categories. The coders were also instructed to assess the overall valence of each respondent's statements - the first of our dependent measures - rating them as: -1 (entirely negative comments); 0 (mixed, including both positive and negative comments); or 1 (entirely positive comments). Following standard content analysis procedures, we tested inter-coder agreement on approximately 50 statements, making sure that a full range of possible types of coding decisions were required of the coders. To assess reliability, we used Krippendorff's alpha [24, 25], a conservative measure that corrects for chance agreement among coders; a K-alpha of .70 or higher is considered sufficient and .80 or higher is considered excellent. For 7 of the 8 thematic categories, we achieved a reliability of .80 or higher; "Lack of Evidence or Stylistically Confusing" was the exception, with an inter-coder reliability of .70. After establishing reliability, the two coders then went on to categorize the rest of the remaining statements from the sample of respondents.

To code the respondent's sentence-specific reactions made with the highlighting pens, sentences marked with only green on at least one word were scored +1 (i.e. indicating "especially clear or useful"), sentences marked with only pink on at least one word were scored -1 (i.e. indicating "especially confusing or unhelpful), and sentences with either no highlighting, or both green and pink, were scored 0. Composite scores were created for each of the four sections of the essay - the opening, the threat section, the benefit section, and the conclusion - by summing the sentence-specific scores in the section and dividing by the number of sentences. A composite score for the entire essay - the second of the dependent measures in our hypothesis - was created by summing the sentence scores across each segment and dividing by the number of respondents per segment. Population estimates, which can be taken solely as preliminary indicators given the non-probabilistic nature of our sampling, were estimated by weighting the mean values for each of the six segments according to its prevalence in the U.S. population (see Figure 1).

Data Analysis

To test the between-segment differences in our dependent measures - overall reactions to the essay (i.e., valence) and composite sentence-specific reactions to the entire essay - we used the nonparametric Kruskal-Wallis test (see Figures 2, 3). To test if the median response to the essay on each dependent measure was greater than zero (i.e., a positive reaction) for our full sample, we used the Wilcoxon signed rank test. Lastly, for both dependent measures, we used the Wilcoxon signed rank test to test our hypothesis that five of the six segments (the Dismissive being the one exception) would respond positively to the essay; the null hypothesis was that the median score for each of the five segments did not differ from zero. The Wilcoxon signed rank test is appropriate for small sample sizes and non-normal distributions, both of which are the case for at least some segments in our data.

Figure 2 Average valence of respondents' general essay comments. The mean valence of respondent comments when asked their general reactions to the public health essay by audience segment and by a national population estimate. Note: 1 = (entirely positive comments); 0 = (mixed, including both positive and negative comments); and -1 = (entirely negative comments). Full size image

Figure 3 Composite essay scores by segment. Scores reflect respondent average values by segment for the difference between the number of times each of 18 sentences were marked "especially clear or helpful" and "especially confusing or unhelpful" with a full range of possible values between 18 and -18. The scores are adjusted for unequal numbers of respondents within each segment by re-weighting values to represent n = 10. Full size image

Post-hoc - after examining the visualized data (see Figures 4, 5 and 6) - we decided to test for two possible main effects in the data. To examine the possibility that the essay's later focus on the public health benefits of mitigation-related policy actions was seen by respondents as clearer and more useful than the essay's earlier focus on public health-related threats, we calculated the difference between the re-scaled (by a factor of 10) average response to both the benefit and the threat sections and then used the Wilcoxon signed rank test to test, by segment, whether the median of these differences was greater than zero. We then evaluated the overall main effect of the essay - across all segments - using the weighted t-test on the differences with weights corresponding to the frequencies of the segments in the population.

Figure 4 Essay evaluations by sentence: Alarmed, Concerned and Cautious segments. Sentence-specific evaluations of the public health essay by respondents in the Alarmed, Concerned and Cautious segments and by a national population estimate. Note: Scores reflect the difference between the number of times a sentence was marked as "especially clear or helpful" and the number of times it was marked as "especially confusing or unhelpful," adjusting for unequal numbers of respondents within each segment by re-weighting values to represent n = 10. Sentence abbreviations correspond to O = opening section (5 sentences); T = climate change health threat related section (7 sentences); B = mitigation-related policy actions and their health benefits (4 sentences); and C = concluding section (2 sentences). The national population estimate was created by weighting the values for each of the six segments according to their relative proportion of American adults. Full size image

Figure 5 Essay evaluations by sentence: Disengaged, Doubtful and Dismissive segments. Sentence-specific evaluations of the public health essay by respondents in the Disengaged, Doubtful and Dismissive segments and by a national population estimate. Note: Scores reflect the difference between the number of times within a sentence was marked as "especially clear or helpful" and the number of times it was marked as "especially confusing or unhelpful," adjusting for unequal numbers of respondents within each segment by re-weighting values to represent n = 10. Sentence abbreviations correspond to O = opening section (5 sentences); T = climate change health threat related section (7 sentences); B = mitigation-related policy actions and their health benefits (4 sentences); and C = concluding section (2 sentences). The national population estimate was created by weighting the values for each of the six segments according to their relative proportion of American adults. Full size image

Figure 6 Essay evaluations by section (opening, threat, benefits, closing). Average section-specific evaluations of the public health essay by respondents in each of the six audience segments and by a national population estimate. Note: Scores reflect the difference between the number of sentences within each section marked by a respondent as "especially clear or helpful" and those marked as "especially confusing or unhelpful" with those values averaged across the number of sentences per section and rescaled by a factor of 10. Section abbreviations correspond to O = opening section (5 sentences); T = climate change health threat related section (7 sentences); B = mitigation-related policy actions and their health benefits (4 sentences); and C = concluding section (2 sentences). The national population estimate was created by weighting the mean values for each of the six segments according to their relative proportion of American adults. Full size image

Lastly, to examine for the possibility that the concluding framing section of the essay was perceived by respondents as clearer and more useful than the opening framing section, we calculated the difference between the re-scaled average response to both the opening and the concluding sections and then used the Wilcoxon signed rank test to test, by segment, whether the median of these differences was greater than zero. We then evaluated the overall main effect - across all segments - using the weighted t-test on the differences with weights corresponding to the frequencies of the segments in the population.